As is known in the art, there exists a class of networks referred to as neural networks which model the behavior of certain human functions. Electronic neural networks have been used to implement mathematical or engineering abstractions of biological neurons. Circuits emulating biological neurons are typically implemented using digital circuits that operate up to a million times faster than actual neurons or with software which simulates the behavior of a biological neuron. One problem with the digital circuit approach, however is that it does not utilize life-like principles of neural computation. Furthermore, a biological nervous system contains thousands or millions of interconnected neurons and thus the complexity of a biological nervous system results in a complex digital circuit.
Similarly, given the complexity of the biological systems, software simulations can take many hours or days even using presently available state-of the-art processing systems. Thus software systems are not appropriate for use in applications which require real time or close to real time performance from such systems.
An electronic circuit that emulates the analog behavior of actual biological neurons, on the other hand, can perform simulations in real time. Thus, to overcome the above limitations with systems implemented using digital circuits or software, electronic circuit neural networks which use principles of neural computation which are more life-like than the digital circuit or software approaches have been developed.
This type of neural network interacts with real-world events in a manner which is the same as or similar to biological nervous systems and can be utilized in a variety of systems including but not limited to electronic and electromechanical systems, such as artificial vision devices and robotic arms. Such neural networks can also be used as research tools to better understand how biological neural networks communicate and learn.
Much of the effort directed toward producing electronic implementations of biological neurons have focused on emulating the input-output functional characteristics of the neuron, essentially treating the neuron as an abstracted black box. These implementations focus on circuits and techniques for generating an action potential in an attempt to simulate the actions neurons take to communicate with one another. One problem with past approaches, however, is that such approaches fail to properly take into account or model the means which actually produces the action potential in a biological neuron.
Some prior art techniques have produced analog integrated circuits that mimic the functional characteristics of real neuron cells, by isomorphically emulating the membrane conductances within an actual neuron cell body. Thus, one problem with prior art approaches is that they fail to include circuitry for the synapse through which neurons communicate and/or the prior art approaches fail to include circuitry for the dendrite which is the connection between the synapse and neuron cell body. Prior art systems also fail to include effective circuitry to implement the adaptation or learning functions of real neurons.
As known to one of ordinary skill in the art, a neuromorphic system emulates the functionality and organization of a biological nervous system on an integrated circuit. Neuromorphic systems typically include analog electronic circuits with digital circuitry to enhance and support their function. Fabrication of these neuromorphic circuits is most often done in a complementary-metal-oxide-semiconductor (CMOS) process using very large-scale integration (VLSI) technology.
Neuromorphic systems directly embody in the physics of their analog CMOS circuit building blocks so-called isomorphisms of the biophysical processes. Neural computational primitives like amplifying, exponentiating, thresholding, integrating, taking the sigmoidal function of, and storing charge, can thus be efficiently performed in a real-time analog fashion using compact low-power CMOS circuits designed specifically for these purposes.
Digital computing paradigms in other fields of science and engineering has led to their use in simulating nervous systems. Digital simulations by themselves, however, require huge and complex Boolean logic functions encode fundamentally analog neural computational primitives, such as those mentioned above. Thus, the modeled system must be translated into an explicitly mathematical form. This is grossly inefficient in terms of time and number of transistors required to execute a neural computation. Significantly, the natural temporal relationship between neuronal processes is not preserved in a digital simulation on a computer thereby preventing real time interaction with the real world in a manner analogous to that of biological nervous system.
Animal nervous system are capable of learning and remembering. One simple type of learning involves the interaction of two neurons, as shown in FIG. 1. Learning occurs when there is an alteration of the synaptic transmission strength from the presynaptic neuron's axon terminal to the postsynaptic neuron's synapse head. A synapse whose strength can be modified by neuronal activity is said to be plastic, and the general phenomenon is known as synaptic plasticity. When neuronal activity leads to an increase in synaptic transmission strength, the synapse is said to have become potentiated. And when this stimulation leads to a decrease in strength, the synapse is said to have become depressed. If these changes are subsequently retained, the “learned” information is “remembered” by the synapse. Potentiation that is retained for a long period of time after neuronal activity has ceased is known as long-term potentiation (LTP). Likewise, depression that is retained for a long period of time after neuronal activity has ceased is known as long term depression (LTD). Both of these phenomena have been shown to occur in the various regions of the brain.
One known electrical model that attempts to explain the biophysical behavior of a Hebbian synapse is shown in FIG. 2. Examining the simple two-neuron system of FIG. 1, when the presynaptic neuron fires an action potential, its axon terminal release neurotransmitters. These neurotransmitters pass through the synaptic cleft and bind to receptors on the synapse head. This causes NMDA and non-NMDA ion channels to open up. The NMDA ion channel passes an electric current, which consists primarily of Ca2+ ions. This postsynaptic influx of Ca2+ ions plays a pivotal role in the expression of synaptic plasticity. Upon entering the synapse head, Ca2+ ions set in motion a series of events that ultimately leads to the induction and maintenance of LTP and/or LTD.
FIG. 3 shows how the level of calcium concentration that has accumulated inside of a synapse operates to change the long-term plasticity. The non-NMDA ion channels pass a current which, in contrast to the NMDA channels, consists mainly of Na+ ions (with a negligible Ca2+ component). The total synaptic current that flows through the membrane thus consists of the sum of (1) the NMDA current, and (2) the non-NMDA current; plus (3) a small leakage current, and (4) a capacitive current that flows when the head membrane voltage is changing.
The leakage conductance ghead is constant while the non-NMDA conductance gnon-NMDA is dependent on the time that elapses after an action potential excites the synapse. It is given by the following alpha-function
            g              non        -        NMDA              ⁡          (      t      )        =      κ    ⁢                  ⁢          g      p        ⁢    t    ⁢                  ⁢          exp      ⁡              (                              -            t                                t            p                          )            where κ=e/tp, e is the base of the natural logarithm, tp=1.5 ms, and the peak conductance gp=0.5 nS. The concentration of calcium within the synapse also modulates this conductance, and that this is the biophysical mechanism by which LTP and LTD are expressed.
Like the non-NMDA conductance gnon-NMDA, the NMDA conductance gNMDA also depends on the time that elapses after an action potential excites the synapse. However, there is an additional dependence on the synapse head membrane voltage Vhead, and there is no dependence on calcium concentration. In this case, the conductance is a sigmoidal function
            g      NMDA        ⁡          (      t      )        =                    exp        ⁡                  (                                    -              t                                      τ              1                                )                    -              exp        ⁡                  (                                    -              t                                      τ              2                                )                            1      +                        η          ⁡                      [                          Mg                              2                +                                      ]                          ⁢                  exp          ⁡                      (                                          -                γ                            ⁢                                                          ⁢                              V                head                                      )                              where τ1=80 msec, τ2=0.67 msec, η=0.33/mM, γ=0.06 mV, and gη=0.2 nS. When Vhead is near its resting potential, the NMDA ion channel conductance is close to zero and little Ca2+ enters the cell. Excitation by action potentials from the presynaptic neuron, however, causes the conductance of the non-NMDA ion channels to increase. This allows an influx of Na+ ions into the synapse head which charges up the membrane capacitance Ch and increases the membrane voltage Vhead. The increase in Vhead in turn causes the conductance of the NMDA channels to rise from zero, allowing an influx of calcium ions that induces LTP and/or LTD or neither, as shown in FIG. 3.
Three broad types of LTP and LTD may be distinguished: hemosynaptic, associative, and heterosynaptic. Homosynaptic LTP and homosynaptic LTD occur in isolated synapses, such as the 2-neuron system of FIG. 1. Homosynaptic LTP is induced when a single synapse is subjected to a burst of high frequency action-potential stimulation from a presynaptic neuron. This type of LTP can be thought of as a sort of microscopic “practice makes perfect.” That is, memory is reinforced through repeated use of the synapse.
Homosynaptic LTD, on the other hand, occurs when the synapse is subjected to a long period of sustained low frequency stimulation. While it may not be intuitive that repeated use of a synapse even at low frequencies result in a synaptic depression, the biological significance makes sense if one considers this sort of LTD to be a microscopic “getting so use to something you forget about it”. As an example, consider the buzzing of fluorescent lights. To someone not used to working in a room with them, they can be quite distracting. But after awhile, this sensitivity disappears.
FIG. 4 summarizes the experimentally determined long-term plasticity behavior of a synapse, e.g. the simple two neuron system of FIG. 1, when it is subjected to a range of presynaptic action potential frequencies. The synaptic strength is a result of the operation of Ca2+ within the synapse, as described above.
It would, therefore, be desirable to provide a neuromorphic circuit that emulates homosynaptic long term potentiation and long term depression.